Picture distribution system and method, picture distribution apparatus and a method therefor, picture receiving apparatus and a method therefore, and recording medium and program used therewith

ABSTRACT

A picture transmitting apparatus transmits, to a picture receiving apparatus, a picture as a predetermined area of an omnidirectional picture. The picture receiving apparatus displays a portion of the transmitted picture which corresponds to a viewpoint. When the viewpoint is moved, information of the movement is transmitted from the picture receiving apparatus to the picture transmitting apparatus, and a picture corresponding to the new viewpoint is transmitted from the picture transmitting apparatus to the picture receiving apparatus. The picture receiving apparatus displays a portion of the received picture which corresponds to the viewpoint.

BACKGROUND OF THE INVENTION

[0001] The present invention relates to picture distribution systems andmethods, picture distributing apparatuses and methods therefor, picturereceiving apparatuses and methods therefor, and recording media andprograms used therewith, and in particular, to a picture distributionsystem and method, a picture distribution apparatus and a methodtherefor, and a picture receiving apparatus and a method therefore inwhich pictures can be distributed without overloading a network, and arecording medium and a program which are used therewith.

[0002] A new picture-space service is in the process of being launched.In the new picture-space service, from pictures having variousviewpoints, such as omnidirectional pictures, a user can selectivelyview a picture from a user-preferred viewpoint. Multi-viewpoint picturessuch as omnidirectional pictures have large amounts of data since theyinclude pictures constituting the space, other than a picture (or adisplayed picture) being actually viewed by the user.

[0003] When the streaming distribution of the multi-viewpoint picturesis performed by a transmission link such as the Internet, in theexisting streaming distribution systems, such as REALSERVER® ofRealVideo and QUICKTIME® of Apple, all pictures constituting the picturespace are distributed from a server to a client. This is because anapplication such as transmission of omnidirectional pictures is notassumed.

[0004] Japanese Unexamined Patent Application Publication No. 11-205772describes the network distribution of omnidirectional pictures. In theinvention in this Publication, the transmitting side transmits allpictures, while the receiving side separates a picture to be displayed.In this type of distribution method, there is a large load on networkbands since all the pictures are distributed despite that most of thepictures are not displayed. Also, when the displayed pictures havecompressed form, decoding size increases, thus increasing the processingload on the client. This causes a problem in that real-time processingis difficult.

[0005] Accordingly, an approach is required in which only the necessarypictures are distributed without distributing all the pictures. InJapanese Unexamined Patent Application Publication No. 2000-132674, asystem is proposed in which a viewpoint picture of omnidirectionalpictures is transmitted and received. In the proposed system, a clientuses coordinates to designate a viewpoint picture, and a servertransmits only a picture at the designated coordinates. The proposedsystem is based on the assumption that it is mainly used for stillpictures. Thus, in response to a user's urgent viewpoint-moving request,a picture from the requested viewpoint cannot be provided due to a delayin response on the network, so that a smooth viewpoint movement is notrealized.

SUMMARY OF THE INVENTION

[0006] The present invention is made in view of the above circumstances,and an object thereof is to distribute pictures in which a viewpoint canbe smoothly moved, without overloading a network and a receiving side.

[0007] According to an aspect of the present invention, a picturedistribution system is provided which includes a transmitting apparatus,and a receiving apparatus. The transmitting apparatus distributes anomnidirectional picture to the receiving apparatus by a network. Thetransmitting apparatus includes a first receiving unit for receivingviewpoint information from the receiving apparatus, an extracting unitfor, based on the received viewpoint information, extracting apredetermined area as a first picture from the omnidirectional picture,and a first transmitting unit for transmitting, to the receivingapparatus, unit regions obtained by dividing the first picture. Thereceiving apparatus includes an acquiring unit for acquiring theviewpoint information, a second transmitting unit for transmitting theacquired viewpoint information to the transmitting apparatus, a secondreceiving unit for receiving the first picture transmitted from thesecond transmitting apparatus, a storage unit for storing the firstpicture received by the second receiving unit, and an output unit foroutputting a second picture as a predetermined area extracted from thestored first picture, the second picture corresponding to the acquiredviewpoint information.

[0008] According to another aspect of the present invention, a picturedistribution method for distributing an omnidirectional picture from atransmitting apparatus to a receiving apparatus by a network isprovided. The transmitting apparatus performs a first receiving step forreceiving viewpoint information from the receiving apparatus, anextracting step for, based on the received viewpoint information,extracting a predetermined area as a first picture from theomnidirectional picture, and a first transmitting step for transmitting,to the receiving apparatus, unit regions obtained by dividing the firstpicture. The receiving apparatus performs an acquiring step foracquiring the viewpoint information, a second transmitting step fortransmitting the acquired viewpoint information to the transmittingapparatus, a second receiving step for receiving the first picturetransmitted from the transmitting apparatus, a storage step for storingthe first picture received in the second receiving step, and an outputstep for outputting a second picture as a predetermined area extractedfrom the stored first picture, the second picture corresponding to theacquired viewpoint information.

[0009] According to another aspect of the present invention, a picturedistribution apparatus for distributing an omnidirectional picture to areceiving apparatus by a network is provided. The picture distributingapparatus includes a receiving unit for receiving viewpoint informationfrom the receiving apparatus, an extracting unit for, based on thereceived viewpoint information, extracting a predetermined area as apicture from the omnidirectional picture, and a transmitting unit fortransmitting, to the receiving apparatus, predetermined unit regionsobtained by dividing the picture. Preferably, based on a motion vectorincluded in the received viewpoint and a round trip time of the network,the extracting unit determines the next predetermined area to beextracted from the omnidirectional picture.

[0010] The picture distribution apparatus may further include acalculating unit for calculating the margin of a predetermined areawhich is output by the receiving apparatus in response to the viewpointinformation, and which corresponds to the predetermined area transmittedto the receiving apparatus by the transmitting unit. Based on thereceived viewpoint information and the calculated margin, the extractingunit may extract the predetermined area as the picture from theomnidirectional picture.

[0011] According to another aspect of the present invention, a picturedistribution method for a picture distribution apparatus fordistributing an omnidirectional picture to a receiving apparatus by anetwork is provided. The picture distribution method includes the stepsof receiving viewpoint information from the receiving apparatus,extracting, based on the received viewpoint information, a predeterminedarea as a picture from the omnidirectional picture, and transmitting, tothe receiving apparatus, predetermined unit regions obtained by dividingthe picture.

[0012] According to another aspect of the present invention, a firstrecording medium containing a computer-readable program for a picturedistribution apparatus for distributing an omnidirectional picture to areceiving apparatus is provided. The program includes the steps ofreceiving viewpoint information from the receiving apparatus,extracting, based on the received viewpoint information, a predeterminedarea as a picture from the omnidirectional picture, and transmitting, tothe receiving apparatus, predetermined unit regions obtained by dividingthe picture.

[0013] According to another aspect of the present invention, a firstprogram for a computer controlling a picture distribution apparatus fordistributing an omnidirectional picture by a network is provided. Theprogram causes the computer to execute the steps of receiving viewpointinformation from the receiving apparatus, extracting, based on thereceived viewpoint information, a predetermined area as a picture fromthe omnidirectional picture, and transmitting, to the receivingapparatus, predetermined unit regions obtained by dividing the picture.

[0014] According to another aspect of the present invention, a picturereceiving apparatus for receiving an omnidirectional picture distributedfrom a transmitting apparatus by a network is provided. The picturereceiving apparatus includes an acquiring unit for acquiring viewpointinformation, a transmitting unit for transmitting the acquired viewpointinformation to the transmitting apparatus, a receiving unit forreceiving a first picture as a predetermined area extracted from theomnidirectional picture, the first picture being transmitted from thetransmitting apparatus, a storage unit for storing the-received firstpicture, and an output unit for outputting a second picture as apredetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information.

[0015] The picture receiving apparatus further includes a calculatingunit for calculating the margin of the second picture which correspondsto the first picture. Based on the calculated margin, the transmittingunit may transmit the viewpoint information to the transmittingapparatus.

[0016] According to another aspect of the present invention, a picturereceiving method for a picture receiving apparatus for receiving anomnidirectional picture distributed from a transmitting apparatus by anetwork is provided. The picture receiving method includes the steps ofacquiring viewpoint information, transmitting the acquired viewpointinformation to the transmitting apparatus, receiving a first picture asa predetermined area extracted from the omnidirectional picture, thefirst picture being transmitted from the transmitting apparatus, storingthe received first picture, and outputting a second picture as apredetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information.

[0017] According to another aspect of the present invention, a secondrecording medium containing a computer-readable program for a picturereceiving apparatus for receiving an omnidirectional picture distributedfrom a transmitting apparatus by a network is provided. The programincludes the steps of acquiring viewpoint information, transmitting theacquired viewpoint information to the transmitting apparatus, receivinga first picture as a predetermined area extracted from theomnidirectional picture, the first picture being transmitted from thetransmitting apparatus, storing the received first picture, andoutputting a second picture as a predetermined area extracted from thestored first picture, the second picture corresponding to the acquiredviewpoint information.

[0018] According to another aspect of the present invention, a secondprogram for a computer controlling a picture receiving apparatus forreceiving an omnidirectional picture distributed from a transmittingapparatus by a network is provided. The program causes the computer toexecute the steps of acquiring viewpoint information, transmitting theacquired viewpoint information to the transmitting apparatus, receivinga first picture as a predetermined area extracted from theomnidirectional picture, the first picture being transmitted from thetransmitting apparatus, storing the received first picture, andoutputting a second picture as a predetermined area extracted from thestored first picture, the second picture corresponding to the acquiredviewpoint information.

[0019] According to a picture distribution system and method of thepresent invention, viewpoint information is transmitted from a receivingapparatus to a transmitting apparatus, and based on the viewpointinformation, a predetermined area is extracted as a first picture froman omnidirectional picture. The first picture is divided into unitregions and is transmitted from the transmitting apparatus to thereceiving apparatus.

[0020] According to a picture distribution apparatus and method, and arecording medium and a program which are used therewith, based onviewpoint information received from a receiving apparatus, apredetermined area is extracted as a picture from an omnidirectionalpicture. The picture is divided into predetermined unit regions, and theregions are transmitted to the receiving apparatus.

[0021] According to a picture receiving apparatus and method, and arecording medium and a program which are used therewith, viewpointinformation is transmitted to a transmitting apparatus, and based on theviewpoint information, a first picture transmitted from the transmittingapparatus is stored, and a predetermined area is extracted as the storedfirst picture and is output.

[0022] Additional features and advantages of the present invention aredescribed in, and will be apparent from, the following DetailedDescription of the Invention and the Figures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0023]FIG. 1 is a block diagram showing an omnidirectional picturedistribution system to which the present invention is applied;

[0024]FIG. 2 is an illustration of an operation of the omnidirectionalpicture distribution system shown in FIG. 1 when a viewpoint is changed;

[0025]FIG. 3 is an illustration of principles of viewpoint movementcontrol;

[0026]FIG. 4 is an illustration of RTT measurement processing based onRTCP;

[0027]FIG. 5 is an illustration of picture division into tiles;

[0028]FIG. 6 is an illustration of state changes of RTSP;

[0029]FIG. 7 is an illustration of state changes of RTSP when the methodMOVEPLAY is added;

[0030]FIG. 8 is a flowchart illustrating a process performed in the INITstate of a picture transmitting apparatus 1;

[0031]FIG. 9 is a flowchart illustrating a process performed in theREADY state of the picture transmitting apparatus 1;

[0032]FIG. 10 is a flowchart illustrating a process performed in thePLAY state of the picture transmitting apparatus 1;

[0033]FIG. 11 is a flowchart illustrating a process performed in theMOVEPLAY state of the picture transmitting apparatus 1;

[0034]FIG. 12 is an illustration of state changes in a picture receivingapparatus 2;

[0035]FIG. 13 is a flowchart illustrating a process performed in theINIT state of the picture receiving apparatus 2;

[0036]FIG. 14 is a flowchart illustrating a process performed in theREADY state of the picture receiving apparatus 2;

[0037]FIG. 15 is a flowchart illustrating a process performed in thePLAY state of the picture receiving apparatus 2;

[0038]FIG. 16 is a flowchart illustrating a process performed in theMOVEPLAY state of the picture receiving apparatus 2; and

[0039]FIG. 17 is a flowchart illustrating a process performed in theMOVE state of the picture receiving apparatus 2.

DETAILED DESCRIPTION OF THE INVENTION

[0040] In the present invention, a distribution method is employed inwhich, when omnidirectional pictures are distributed through a network,by transmitting a picture corresponding to a viewpoint selected by auser, and the periphery, a response can be made to an urgentviewpoint-moving request from the user. The present invention isspecifically described below with reference to the accompanyingdrawings.

[0041] At first, the configuration and operation of the entire system ofthe present invention are described with reference to FIGS. 1 to 4.After that, more detailed points are explained with reference to FIGS. 5to 17.

[0042]FIG. 1 is a block diagram showing an omnidirectional picturedistribution system to which the present invention is applied. Theomnidirectional picture distribution system includes a picturetransmitting apparatus 1 which generates a multi-viewpoint picturestream such as omnidirectional pictures, and at least a picturereceiving apparatus 2 which receives the picture stream from the picturetransmitting apparatus 1. The picture transmitting apparatus 1 and thepicture receiving apparatus 2 are connected to each other by a network 3including the Internet, a local area network (LAN), a wide area network(WAN), etc.

[0043] The picture transmitting apparatus 1 includes a picture inputunit 11, a stream control unit 12, a viewpoint-information receivingunit 13, and a stream transmitting unit 14. A plurality of picturereceiving apparatuses 2 are connected to the picture transmittingapparatus 1 by the network 3.

[0044] The picture input unit 11 includes, for example, a videocamera,and a disk device storing picture files. A picture input from thepicture input unit 11 is transferred to the stream control unit 12. Thepicture transferred to the stream control unit 12 may be acompression-coded picture stream.

[0045] The viewpoint-information receiving unit 13 receives viewpointinformation from the picture receiving apparatus 2, and includes anetwork interface. The viewpoint information from theviewpoint-information receiving unit 13 and the picture stream from thepicture input unit 11 are input to the stream control unit 12, and itgenerates a stream suitable for a viewpoint picture for the picturereceiving apparatus 2.

[0046] The picture stream output from the stream control unit 12 may bea plurality of streams generated by spatially diving a picture into aplurality of blocks. A method for the division into blocks is, forexample, tile division in JPEG (Joint Photographic Experts Group) 2000.The details are described later with reference to FIG. 5.

[0047] The stream transmitting unit 14 uses the Real-time TransportProtocol (RTP) or the like to convert the picture stream into packets,and transmits the packets to the picture receiving apparatus 2. Thestream transmitting unit 14 also includes a network interface.

[0048] The picture receiving apparatus 2 includes aviewpoint-information input unit 21, a display-picture control unit 22,a viewpoint-information transmitting unit 23, a stream receiving unit24, and a picture output unit 25. The viewpoint-information input unit21 has, for example, input devices such as a mouse, a keyboard, a headtracker, and a joystick. Viewpoint information input from theviewpoint-information input unit 21 is transferred to thedisplay-picture control unit 22 and the viewpoint-informationtransmitting unit 23.

[0049] The viewpoint-information transmitting unit 23 transmits theviewpoint information to the picture transmitting apparatus 1, andincludes a network interface. The stream receiving unit 24 includes anetwork interface transfers a received picture stream to thedisplay-picture control unit 22. Based on the viewpoint information, thedisplay-picture control unit 22 generates a picture to be displayed fromthe received picture stream. The display-picture control unit 22 alsoperforms synchronizing processing for the case of receiving a pluralityof picture streams. The picture output unit 25 also includes, forexample, a display device or the like, and outputs and displays thepicture generated by the display-picture control unit 22.

[0050]FIG. 2 shows the communication between the picture receivingapparatus 2 and the picture transmitting apparatus 1 when, in the systemof FIG. 1, the viewpoint in the picture receiving apparatus 2 changes.Based on the viewpoint information received from the picture receivingapparatus 2 through the network 3 and the viewpoint-informationreceiving unit 13, the stream control unit 12 in the picturetransmitting apparatus 1 extracts, from the picture input unit 11, atransmitting picture 202 which is a predetermined area (part) of theentire spatial picture 201 among multi-viewpoint pictures such asomnidirectional pictures. In steps S1, S2, etc., the stream transmittingunit 14 transmits the extracted picture 202 to the picture receivingapparatus 2 through the network 3.

[0051] In the picture receiving apparatus 2, the display-picture controlunit 22 temporarily stores in a built-in storage portion the picture 202received through the stream receiving unit 24. The display-picturecontrol unit 22 captures, from the viewpoint-information input unit 21,information (viewpoint information) relating to a viewpoint(coordinates) based on a user's operation, and re-generates, from thestored picture 202, a display picture in the area corresponding to theviewpoint. The display-picture control unit 22 outputs and displays there-generated picture as a user's-viewpoint picture 204 on the pictureoutput unit 25.

[0052] When, in the picture receiving apparatus 2, occurrence of aviewpoint movement in the direction indicated by the arrow 205 is inputto the viewpoint-information input unit 21 based on a user's operation,in step S11, the viewpoint-information transmitting unit 23 notifies thepicture transmitting apparatus 1 of the viewpoint movement by using thenetwork 3. If the viewpoint-information transmitting unit 23 immediatelynotifies the picture transmitting apparatus 1 of the viewpoint movementafter the viewpoint movement occurs, a certain time (hereinafterreferred to as a “response time TD”) is required until the picturereceiving apparatus 2 receives a picture reflecting the viewpointmovement, due to a reason such as a delay in transmission on the network3 and a delay in processing by the picture transmitting apparatus 1.During a period (the response time TD) until the picture reflecting theviewpoint change reaches the picture receiving apparatus 2, thedisplay-picture control unit 22 determines a picture 207 in a newdisplay position matching the direction of the movement, and displaysthe picture 207 on the picture output unit 25.

[0053] In step S3, in the picture transmitting apparatus 1, based on theviewpoint-movement information received through theviewpoint-information receiving unit 13, the stream control unit 12immediately determines a picture 208 in the position of the next pictureto be transmitted, inputs the picture 208 from the picture input unit11, and transmits the picture 208 from the stream transmitting unit 14.

[0054] When receiving the picture 208, which reflects the viewpointmovement, through the stream receiving unit 24, the display-picturecontrol unit 22 in the picture receiving apparatus 2 stores the receivedpicture 208, and determines and extracts, from the stored picture 208, apicture 210 in the position corresponding to the coordinates (viewpoint)input from the viewpoint-information input unit 21. The display-picturecontrol unit 22 outputs and displays the extracted picture 208 on thepicture output unit 25. The above operations are continuously executedby the picture transmitting apparatus 1 and the picture receivingapparatus 2.

[0055]FIG. 3 shows viewpoint movement control using the relationshipbetween received (transmitting) picture size and displayed picture size.As shown in FIG. 2, after the picture receiving apparatus 2 transmits aviewpoint movement notification, the viewpoint movement cannot beimmediately reflected in a received picture. Thus, within the responsetime TD, control must be performed so that a displayed picture 302 isnot positioned out of the frame of a received picture 301. Thedisplay-picture control unit 22 employs, as this control, any one of thefollowing methods.

[0056] In a first method, the upper limit of a viewpoint moving speed inthe picture receiving apparatus 2 is set based on displayed picturemargins (the vertical margin 303 and the horizontal margin 303 in FIG.3) and the response time TD. The vertical margin 303 represents thedistance between a horizontal end (the upper end in the example of FIG.3) of the displayed picture 302 and a horizontal end (the upper end inthe example of FIG. 3) of the received picture 301. The margin 304represents the distance between a vertical end (the right end in FIG. 3)of the displayed picture 302 and a vertical end (the right end in FIG.3) of the received picture 301.

[0057] In this case, the upper limit of the viewpoint moving speed isdefined by the following expression:

(Upper Limit of Viewpoint Moving Speed)<(Picture Margin)/(ResponseTime)  (1)

[0058] When the movement direction is arbitrary, the movement directionis decomposed into a horizontal component and a vertical component forcalculation. For example, as shown in FIG. 3, when the viewpoint movesin the direction indicated by the arrow (motion vector) 305, themovement direction of the motion vector 305 is decomposed into ahorizontal component 307 and a vertical component 306 for calculation.When the angle between the movement direction and the horizontaldirection is represented by θ, the following expression holds:

(Moving Speed in θ Direction)≦Min(Horizontal Component 304/cosθ,Vertical Component 303/sinθ)/Response Time  (2)

[0059] where Min (A, B) represents the minimum between A and B, and thehorizontal component 304 and the vertical component 303 represent theabsolute values of lengths, respectively. Similarly, expression (2) canbe applied to viewpoint zooming-in or zooming-out by assuming that thedisplayed picture moves in four directions.

[0060] In a second method, the transmitting picture size is set based onthe response time, with the viewpoint moving speed regarded as constant.In this case, transforming expression (1) gives the followingexpression:

(Picture Margin)≧(Viewpoint Moving Speed)×(Response Time)  (3)

[0061] Thus, the transmitting picture size is represented by thefollowing expression:

(Transmitting Picture Size)≧(Viewpoint Moving Speed)×(ResponseTime)+(Displayed Picture Size)  (4)

[0062] In a third method, the number of pixels is set based on theresponse time, with the transmission picture size and the viewpointmoving speed regarded as constant. By transforming expression (4), whilepaying attention to the following points:

[0063] displayed picture size=the number of displayed pixels; andtransmitting picture size=received picture size, the followingexpression is obtained:

(Number of Displayed Pixels)≦(Received Picture Size)−(Viewpoint MovingSpeed)×(Response Time)  (5)

[0064] In this case, it is possible that the third method be combinedwith processing such as enlargement by digital zooming of superficialdisplayed-picture size for a reduction in the number of displayedpixels.

[0065] In a fourth method, the above first to third methods are flexiblyswitched in proportional to the length of the response time. In the caseof a long response time, there is a possibility that the first methodmay cause a very delayed viewpoint movement. Accordingly, a method ofswitching to the second method is possible.

[0066] The response time used in the fourth method is shown as follows:

Response Time=Buffering Time in Picture Receiving Apparatus 2+RoundTrip(Transmission Delay)Time on Network(RTT)+Processing Time in PictureTransmitting Apparatus 1

[0067] Each value is made known by exchanging monitoring informationbetween the picture transmitting apparatus 1 and the picture receivingapparatus 2.

[0068] Also, the RTT included in the response time can be measured bythe following manner. Measurement of an RTT using Real-time ControlProtocol (RTCP) uses a sender report (SR) which is periodicallytransmitted from the picture transmitting apparatus 1 to the picturereceiving apparatus 2 and a receiver report (RR) with which the picturereceiving apparatus 2 responds to the picture transmitting apparatus 1.Each of the SR and the RR is one of messages in RTCP. In the SR, itstransmission time is written, and in the RR, the transmission timewritten in the SR and a time up to transmission of the RR afterreceiving the SR are written.

[0069]FIG. 4 shows that the picture transmitting apparatus 1 and thepicture receiving apparatus 2 exchange an SR and an RR. At time T1, thepicture transmitting apparatus 1 transmits an SR 401 to the picturereceiving apparatus 2. Then, in the SR 401, the transmission time T1 iswritten. The display-picture control unit 22 in the picture receivingapparatus 2 receives the SR 401 at time T2, calculates a one-way delaytime (RTT/2) by using the following expression:

One-way Delay Time (RTT/2)=T2−T1  (6)

[0070] and generates an RTT by doubling the one-way delay time.

[0071] This processing is based on the assumption that the internalclocks of the picture transmitting apparatus 1 and the picture receivingapparatus 2 synchronize with each other. Although a method forestablishing the synchronization between the internal clocks is notdescribed, the synchronization can be established by using Network TimeProtocol (NTP) which is commonly used in the Internet, or an originalprotocol for NTP.

[0072] After a predetermined time (“delay”) elapses, at time T3, thepicture receiving apparatus 2 transmits the RR 402. Then, in the RR 402,the transmission time Ti and a time (delay=T3−T2) from reception of theSR 401 up to transmission of the RR 402 are written. The stream controlunit 12 in the picture transmitting apparatus 1 receives the RR 402 atthe time T4, and calculates an RTT by using the following expression:

RTT=T4−T1−delay=T4−T1−(T3−T2)  (7)

[0073] In the calculation of the RTT by the picture transmittingapparatus 1, the synchronization between the internals clocks of thepicture transmitting apparatus 1 and the picture receiving apparatus 2does not always need to be established because the RR includes the“delay” time. The method in expression (7) is also described in RFC(Request for comments) 1889.

[0074] As described above, by using the present invention, whenmulti-viewpoint pictures, such as omnidirectional pictures, aredistributed by the network 3, bands used on the network 3 and theprocessing load in a receiving terminal can be reduced, so thatpicture-spatial service on a basis of omnidirectional pictures can beprovided even to relatively narrow band networks such as wirelessnetworks and powerless terminals such as personal digital assistants(PDAs) and cellular phones.

[0075] In a case in which the pictures are based on Motion JPEG (JointPhotographic Experts Group) 2000, when a partial picture is distributed,it is not necessary to separately distribute information indicating thatthe distributed picture corresponds to which of the entire picturebecause JPEG 2000 has a tile division function as an encoding functionfor dividing a screen into partial screens, so that the protocol betweenthe transmitter and the receiver can be simplified, thus enablingfacilitated control of viewpoint movement.

[0076] Next, a specific example for realizing the above-described systemis described. At first, the data format of omnidirectional pictures aredescribed. The picture input unit 11 converts the data format ofomnidirectional pictures into a cylindrical picture format. The pictureinput unit 11 uses JPEG 2000 as a picture compressing method. The use ofJPEG 2000 produces the following advantages.

[0077] JPEG 2000 is suitable for transmission of omnidirectionalpictures because it does not need the difference between image frames.As in omnidirectional pictures, when picture size constituting theentire space is very large, a partial picture in an arbitrary viewpointdirection is only shown, and the viewpoint is moved at random by theuser, prediction encoding used in MPEG or the like is almost useless.Also, when considering transmission on the Internet, MPEG which needsinterframe difference has a problem in that, when a packet loss occurs,an error is transmitted to the subsequent frames. This problem does notoccur in JPEG 2000 since it does not need interframe difference.

[0078] In JPEG 2000, after a picture is divided into a plurality ofarbitrary areas called “tiles”, only some tiles can be decoded. Eachtile size can be set to be constant in a size designated in an encodingmode, and the tile size is described in a header in JPEG 2000. Inaddition, index numbers are assigned to the tiles from the top left tobottom right of the entire picture. Accordingly, the numbering has anadvantage in that knowing a tile index number can easily specify whichportion of the entire picture the tile picture is. Moreover, eachportion can be separately encoded. Thus, by controlling encoding of apicture so that the quality of portions (e.g., the upper end and lowerend of a spatial picture mapped in a cylindrical shape) to which theuser does not pay a lot of attention is reduced to increase acompression factor, more storage area and network bandwidths can besaved.

[0079] As described with reference to FIG. 2, this picture distributionsystem uses a method for distributing a viewpoint picture and a picture(e.g., the image 202 in FIG. 2) which consists of an arbitrary number oftiles and which includes its periphery. Only the required number oftiles can be distributed because JPEG 2000 has a function in which, evenif a truncate image is input to a decoder (the display-picture controlunit 22), it can be decoded. FIG. 5 shows an example in which an imageis divided into ten tiles. The tiles are sequentially numbered 0 to 9 asindex numbers. Accordingly, for example, the coordinates of the top leftof the tile image of index 3 can be easily known as (tile_width×3, 0).As shown in FIG. 5, “tile_width” represents a tile width, and“tile-height” represents a tile height.

[0080] Next, a protocol for use in viewpoint movement is described. In aprotocol, used between the picture receiving apparatus 2 (client) andthe picture transmitting apparatus 1 (server) when the viewpoint in thepicture receiving apparatus 2 changes, the following points areconsidered. Although an example of Real Time Streaming Protocol (RTSP)in extended form is described here, a new original protocol may bedesigned. Since extension of RTSP generates portions in stream controlwhich can be used in common, it is only required to define simplecommands.

[0081] Conditions for consideration are as follows:

[0082] (1) A viewpoint-moving request must be transmitted by using areliable communication link or a communication protocol. Thiscorresponds to the TCP.

[0083] (2) A viewpoint moving direction or a motion vector must betransmitted from the client to the server. The motion vector includesnot only a direction of movement but also representation of a movement(moving speed) per unit time.

[0084] (3) It is preferable for the server to notify the client ofestimated arrival information of a partial picture reflecting theviewpoint-moving request. The estimated arrival information isinformation that the client should know beforehand and that indicatesfrom which packet the coordinates of a received picture will change. Inthe case of using RTP to transmit pictures, the estimated arrivalinformation includes time-stamp information and sequence numbers of RTPpackets. By allowing the client to know the estimated arrivalinformation beforehand, the upper limit of the viewpoint moving speed onthe receiving side can be easily determined.

[0085] (4) It is preferable that a partial picture to be transmitted beeasily re-formed.

[0086] (5) It is preferable that the coordinates of a received partialpicture be easily found.

[0087] Next, a specific example of the protocol is described. Theextended RTSP is described below. The details of RTSP are disclosed inRFC 2326. RTSP controls delivery of real-time data in the applicationlevel, and can use TCP as a transport-layer protocol. This satisfies theabove condition (1).

[0088] RTSP, PLAY, STOP, PAUSE, etc., are defined as stream-controlcommands, but a watching and listening method such as viewpoint movementis not assumed. Accordingly, new commands corresponding to viewpointmovement are defined. RTSP has a lot of extensibility, and new commandscan be easily defined. Command adding techniques can include a techniquefor adding new parameters to conventionally existing methods (PLAY,STOP, PAUSE, etc.), and a technique for adding a new method. Betweenthese, the technique for adding a new method is preferable because a newstate is generated by adding a command. Accordingly, an example of themethod addition is described below. However, the technique for addingnew parameters has an advantage in that mounting is simplified, comparedwith the method addition.

[0089] RTSP has state transition as shown in FIG. 6. By adding aviewpoint moving command, the state transition is changed to that shownin FIG. 7. In FIGS. 6 and 7, the encircled portions indicate states, andthe arrows indicate changes. An explanation of each arrow is a methodname, and the arrow indicates that activation of the method causes astate change.

[0090] By way of example, in the RTSP state transition shown in FIG. 6,by activating the method TEARDOWN in the PLAY state, the state ischanged into the INIT state. Also, by activating the PAUSE state in thePLAY state, the state is changed into the READY state. The INIT state ischanged into the READY state by activating the method SETUP. The READYstate is changed into the INIT state by activating the method TEARDOWN,and is changed into the PLAY state by activating the method PLAY.

[0091] In the RTSP state transition (shown in FIG. 7) to which themethod MOVEPLAY is added, in addition to the state changes shown in FIG.6, state changes occur among the MOVEPLAY state, the INIT state, thePLAY state, and the READY state.

[0092] Specifically, the MOVEPLAY state is changed into the INIT stateby activating the method TEARDOWN, is changed into the PLAY state byactivating the method MOVEPLAY, and is changed into the READY state byactivating the method PAUSE. The PLAY state is changed into the MOVEPLAYstate by activating the method MOVEPLAY.

[0093] In the present invention, the method MOVEPLAY is added. MOVEPLAYis a method designating viewpoint movement, and is sent at the start ofviewpoint movement. Parameters interpreted by the method MOVEPLAY areidentical to those of the method PLAY, but differ from those of themethod PLAY in vector parameters representing viewpoint movement andzoom parameters representing zooming. When the motion vector of theviewpoint or zooming factor does not change, and when viewpoint movementis continuously performed, it is not necessary to sent the methodMOVEPLAY to the server again.

[0094] Next, the vector parameter is described. Occurrence of theviewpoint movement causes the client to send, to the server, thefollowing request:

[0095] MOVEPLAY rtsp://video.example.com/video RTSP/1.0

[0096] Cseq: 835

[0097] Session: 123456

[0098] Vector:xy 10-30

[0099] The first viewpoint-movement stopping request designates a movingspeed of zero.

[0100] The common format of the vector parameter is as follows:

[0101] Vector:xy (Moving Speed in X-axis Direction) [(Moving Speed inY-axis Direction)] or

[0102] Vector:polar (Rotational Angle with X-axis set to 0°) [(MovingSpeed)]

[0103] For example, when the X-Y coordinate system is used fordesignation such as “vector:xy 10-30”, the designation indicates thatviewpoint movement is performed at a speed of 10 pixels in the Xdirection (the right direction) and −30 pixels (i.e., downwardly 30pixels) in the Y direction (the upward direction) per unit time (onesecond) in the X-Y coordinate system. When the moving speed in the Ydirection is zero, the representation for the Y axis may be omitted.

[0104] Also, for example, when the polar coordinate system is used todesignate a motion vector, the designation indicates that viewpointmovement is performed at a speed of 20 pixels in an anticlockwise90-degree direction (upward) per unit time (one second). When theviewpoint moving speed is fixed, designation is easier in the polarcoordinate system than in the X-Y coordinate system. When the movingspeed is fixed, and the agreement between the server and the client isobtained as an application specification or by negotiation at the startof a session, the moving speed may be omitted.

[0105] Next, the zoom parameter is described. The zoom parameter is usedto request zooming in or zooming out, and is treated similarly to thevector parameter. The zoom parameter is shown as follows:

[0106] Zoom:(Zoom Speed)

[0107] In this zoom parameter, a positive value equal to zero or greateris set as the zoom speed. The value 1.0 represents 1× magnificationzooming, a value greater than 1.0 represents zooming in (enlargingdirection), and a value less than 1.0 represents zooming out (reducingdirection). An enlargement factor or a reduction factor is greater as itis further than 1.0. For example, in the case of a zooming speed of 2.0,the speed represents zooming a double magnification per frame, and inthe case of a zooming speed of 3.0, the speed represents zooming at atriple speed per frame. The server responds with an error when a zoomingspeed equal to zero or less is designated, or further zooming isimpossible.

[0108] Next, the method MOVESTOP is described. The method MOVESTOP is acommand designating a viewpoint-movement stop and is used to stopviewpoint movement. Parameters interpreted by the method MOVESTOP aresimilar to those interpreted by the method STOP.

[0109] Next, replies of the server are described. When the serverreceives the method MOVEPLAY or MOVESTOP, and cannot interpret thereceived method, it replies with “Method Not Allowed” as defined in RFC2326. When the server can interpret the method, it replies in accordancewith the method PLAY or STOP. In order to indicate from which sequencethe request is reflected, the replay include the following RTP-Infoheader:

[0110] RTSP/1.0 200 OK

[0111] Cseq:835

[0112] Session:123456

[0113]RTP-Info:url=rtsp://video.examplecompany/video;seq=3456;rtptime=789012

[0114] In order to fulfill the roll of transmitting a picture inresponse to a request from the picture receiving apparatus 2, thepicture transmitting apparatus 1 has state transition similar to that inRTSP. Accordingly, the picture transmitting apparatus 1 has four states,the INIT state, the READY state, the PLAY state, and the MOVEPLAY state.Processes corresponding to the four states are shown in FIGS. 8 to 11.Step S12 (FIG. 8) in the INIT state, steps S51, S52, and S55 (FIG. 10)in the PLAY state, and the entire process (FIG. 11) of the MOVEPLAYstate added steps added by extending the RTSP method.

[0115] As shown in FIG. 8, after the process is started, the streamcontrol unit 12 sets the INIT state in step S11, and executesinitialization of transmitting tile numbers in step S12. In step S13,the stream control unit 12 determines whether the method SETUP has beenreceived. If it has not been received yet, in step S14, the streamcontrol unit 12 determines whether a termination signal has beenreceived. If the termination signal has not been received, the streamcontrol unit 12 returns to step S13, and repeats the subsequent steps.In step S13, if it is determined that the method SETUP has beenreceived, the stream control unit 12 proceeds to step S15, and the stateis changed into the READY state (FIG. 7). In step S14, if it isdetermined that the termination signal has been received, the processends.

[0116] In the READY state shown in FIG. 9, in step S31, the streamcontrol unit 12 determines whether the method PLAY has been received. Ifit has not been received yet, the stream control unit 12 proceeds tostep S32, and determines whether the method TEARDOWN has been received.If it is determined that the method TEARDOWN has not been received, thestream control unit 12 returns to step S31, and repeats the subsequentsteps.

[0117] In step S31, if it is determined that the method PLAY has beenreceived, the stream control unit 12 proceeds to step S33, and changesthe state into the PLAY state (FIG. 7). In step S32, it is determinedthat the method TEARDOWN has been received, the stream control unit 12proceeds to step S34, and changes the state into the INIT state (FIG.7).

[0118] In the PLAY state shown in FIG. 10, in step S51, the streamcontrol unit 12 transmits a selected tile. In step S52, the streamcontrol unit 12 determines whether the method MOVEPLAY has beenreceived. If the method MOVEPLAY has been received, the stream controlunit 12 proceeds to step S55, and changes the state into the MOVEPLAYstate (FIG. 7). In step S52, if it is determined that the methodMOVEPLAY has not been received, the stream control unit 12 proceeds tostep S53, and determines whether the method PAUSE has been received. Ifit is determined that the method PAUSE has been received, the streamcontrol unit 12 proceeds to step S56, and changes the state into theREADY state (FIG. 7).

[0119] In step S53, if it is determined that the method PAUSE has notbeen received, the stream control unit 12 proceeds to step S54, anddetermines whether the method TEARDOWN has been received. If the methodTEARDOWN has been received, the stream control unit 12 proceeds to stepS57, and changes the state into the INIT state (FIG. 7). In step S54, ifit is determined that the method TEARDOWN has not been received, thestream control unit 12 returns to step S51, and repeats the subsequentsteps.

[0120] In the MOVEPLAY state shown in FIG. 11, in step S71, the streamcontrol unit 12 sets RTT in the elapsed time. The RTT is a measuredvalue described with reference to FIG. 4. In step S72, the streamcontrol unit 12 calculates a motion based on the following expression:

Motion=Motion Vector×Elapsed Time

[0121] The value set in step S71 is used as the elapsed time.

[0122] In step S73, the stream control unit 12 executes the process forselecting a transmitting tile again. In step S74, the stream controlunit 12 controls the stream transmitting unit 14 to transmit thetransmitting tile selected in step S73. In step S75, the stream controlunit 12 determines whether the method MOVESTOP has been received. If ithas been received, in step S79, the stream control unit 12 changes thestate into the PLAY state (FIG. 7).

[0123] In step S75, if the method MOVESTOP has not been received, thestream control unit 12 proceeds to step S76, and determines whether themethod PAUSE has been received. If it is determined that the methodPAUSE has been received, the stream control unit 12 proceeds to stepS80, and executes transmission of all the tiles. In step S81, the streamcontrol unit 12 changes the state into the READY state (FIG. 7).

[0124] In step S76, if the method PAUSE has been received, the streamcontrol unit 12 proceeds to step S77, and determines whether theTEARDOWN has been received. If it is determined that the method TEARDOWNhas been received, the stream control unit 12 proceeds to step S80 andchanges the state into the INIT state (FIG. 7).

[0125] In step S77, if it is determined that the method TEARDOWN hasbeen received, the stream control unit 12 proceeds to step S78 and setsthe reciprocal of the frame rate in the elapsed time. After that, thestream control unit 12 returns to step S72, and repeats the subsequentsteps.

[0126] The picture receiving apparatus 2 must respond to an interactiverequest from the user other than the protocol operations with thepicture transmitting apparatus 1. Accordingly, in the picture receivingapparatus 2, the MOVE state, which is a closed state in the picturereceiving apparatus 2, is added to the states in FIG. 7 of thetransmitting side, so that the picture receiving apparatus 2 has statetransition shown in FIG. 12. Each state in FIG. 12 is identical inmeaning to that in FIG. 7. A state change is performed in response to auser's request. The arrows of the state changes are labeled user'srequests.

[0127] In the example in FIG. 12, when a STOP request is made in thePLAY state, the PLAY state is changed into the INIT state. When a PAUSErequest is made, the PLAY state is changed into the READY state. When aviewpoint-moving request is made, the PLAY state is changed into theMOVEPLAY state.

[0128] When a PLAY request is made in the INIT state, the INIT state ischanged into the READY state. When a PLAY request is made in the READYstate, the READY state is changed into the PLAY state. When a STOPrequest is made, the READY state is changed into the INIT state. When aviewpoint-moving request is made, the READY state is changed into theREADY state.

[0129] When a viewpoint stopping request is made in the MOVE state, theMOVE state is changed into the READY state. When a STOP request is madein the MOVEPLAY state, the MOVEPLAY state is changed into the INITstate. When a viewpoint stopping request is made, the MOVEPLAY state ischanged into the PLAY state. When a PAUSE request is made, the MOVEPLAYstate is changed into the READY state.

[0130] FIGS. 13 to 17 show processes for the states. Steps S123, S125,S129, and S130 (FIG. 14) in the READY state, steps S142, S143, S146, andS147 (FIG. 15) in the PLAY state, and the entire processes of theMOVEPLAY state (FIG. 16) and the MOVE state (FIG. 17) are addedprocesses. In the MOVE state, no messages are exchanged with the picturetransmitting apparatus 1.

[0131] Referring to the INIT state in FIG. 13, in step S101, the INITstate is set. In step S102, the display-picture control unit 22determines whether a PLAY request has been made. If it is determinedthat the PLAY request has been made, the process proceeds to step S104,and the display-picture control unit 22 executes transmission of themethod SETUP. In step S105, the display-picture control unit 22 changesthe state into the READY state (FIG. 12).

[0132] In step S102, if it is determined that the PLAY request has notbeen made, the process proceeds to step S103, and the display-picturecontrol unit 22 determines whether a termination signal has beengenerated. If it is determined that the termination signal has not beengenerated, the process returns to step S102, and the subsequent stepsare repeatedly executed. If it is determined that the termination signalhas been generated, the process is terminated.

[0133] In the READY state in FIG. 14, the display-picture control unit22 determines in step S121 whether a PLAY request has been made. If ithas not been made, in step S122, the display-picture control unit 22determines whether a STOP request has been made. If it has not beenmade, the display-picture control unit 22 proceeds to step S123 anddetermines whether a viewpoint-moving request has been made. If it isdetermined that the viewpoint-moving request has also not been made, thesubsequent steps are repeatedly executed.

[0134] In step S121, if it is determined that the PLAY request has beenmade, the display-picture control unit 22 proceeds to step S124 andtransmits the method PLAY. In step S125, the picture receiving apparatus2 executes initialization of a margin value of a received picturecorresponding to a viewpoint picture. After that, the display-picturecontrol unit 22 proceeds to step S126 and changes the state into thePLAY state (FIG. 12).

[0135] In step S122, if it is determined that the STOP request has beenmade, the display-picture control unit 22 proceeds to step S127 andtransmits the method TEARDOWN. In step S128, the display-picture controlunit 22 changes the state into the INIT state (FIG. 12).

[0136] In step S123, if it is determined the viewpoint-moving requesthas been made, the display-picture control unit 22 proceeds to step S129and determines whether all the tiles have been received. If they havenot been received, the display-picture control unit 22 returns to stepS121 and continuously executes the subsequent steps. In step S129, if itis determined that all the tiles have been received, the display-picturecontrol unit 22 proceeds to step S130 and changes the state into theMOVE state (FIG. 12).

[0137] In the PLAY state in FIG. 15, in step S141, the display-picturecontrol unit 22 receives a tile picture. In step S142, thedisplay-picture control unit 22 separates a portion corresponding to theviewpoint picture from the tile picture received and stored in thebuilt-in memory in step S141, and outputs and displays the separatedportion on the picture output unit 25.

[0138] In step S143, the display-picture control unit 22 determineswhether the viewpoint-moving request has been made. If it has not beenmade, the display-picture control unit 22 proceeds to step S144 anddetermines whether the PAUSE request has been made. When determiningthat even the PAUSE request has not been made, the display-picturecontrol unit 22 proceeds to step S145 and determines whether the STOPrequest has been made. If it has not been made, the display-picturecontrol unit 22 returns to step S141, and repeatedly executes thesubsequent steps.

[0139] In step S143, if it is determined that the viewpoint-movingrequest has been made, the display-picture control unit 22 proceeds tostep S146 and executes transmission of the method MOVEPLAY. In stepS147, the display-picture control unit 22 changes the state into theMOVEPLAY state (FIG. 12).

[0140] In step S144, if it is determined that the PAUSE request has beenmade, the display-picture control unit 22 proceeds to step S148 andtransmits the method PAUSE. The display-picture control unit 22executes, in step S149, reception of all the tiles through the streamreceiving unit 24, and changes the state into the READY state in stepS150 (FIG. 12).

[0141] In step S145, if it is determined that the STOP request has beenmade, the display-picture control unit 22 proceeds to step S151, andtransmits the method TEARDOWN. In step S150, the display-picture controlunit 22 changes the state into the INIT state (FIG. 12).

[0142] In the MOVEPLAY state in FIG. 16, when the display-picturecontrol unit 22 receives the picture in step S161, it determines, instep S162, whether, in the picture received in step S161, theviewpoint-moving request is reflected. If it is not reflected, thedisplay-picture control unit 22 proceeds to step S163 and updates amargin value in the moving direction. In other words, this increases orreduces the margin in the moving direction by the motion.

[0143] In step S162, if it is determined that in the received picturethe viewpoint-moving request is reflected, the display-picture controlunit 22 proceeds to step S164 and executes re-calculation of the marginvalue. After steps S163 and S164, the display-picture control unit 22proceeds to step S165. In step S165, the display-picture control unit 22extracts a picture corresponding to the viewpoint input from theviewpoint-information input unit 21, and outputs and displays theextracted picture on the picture output unit 25.

[0144] Proceeding to step S166, the display-picture control unit 22determines whether the motion vector has been changed. If it has notbeen changed, the display-picture control unit 22 returns to step S161and continuously executes the subsequent steps.

[0145] In step S166, if it is determined that the motion vector has beenchanged, the display-picture control unit 22 proceeds to step S167, andexecutes re-calculation of the motion vector. In other words, thedisplay-picture control unit 22 further acquires, from theviewpoint-information input unit 21, the moving direction, andcalculates the moving speed based on the following expression:

Moving Speed=Margin/RTT

[0146] Next, proceeding to step S168, the display-picture control unit22 transmits the method MOVEPLAY. In step S169, the display-picturecontrol unit 22 determines whether the viewpoint stopping request hasbeen made. If it has not been made, the display-picture control unit 22proceeds to step S170 and determines whether the PAUSE request has beenmade. When even the PAUSE request has not been made, the display-picturecontrol unit 22 proceeds to step S171 and determines whether the STOPrequest has been made. If it has not been made, the display-picturecontrol unit 22 returns to step S161 and continuously executes thesubsequent steps.

[0147] In step S169, if it is determined that the viewpoint stoppingrequest has been made, the display-picture control unit 22 proceeds tostep S172 and transmits the method MOVESTOP. In step S173, thedisplay-picture control unit 22 changes the state into the PLAY state(FIG. 12).

[0148] In step S170, if it is determined that the PAUSE request has beenmade, the display-picture control unit 22 proceeds to step S174 andtransmits the method PAUSE. In step S175, the display-picture controlunit 22 executes reception of all the tiles through the stream receivingunit 24. In step S176, the display-picture control unit 22 changes thestate into the READY state (FIG. 12).

[0149] In step S171, if it is determined that the STOP request has beenmade, the display-picture control unit 22 proceeds to step S177 andexecutes transmission of the method TEARDOWN. In step S178, thedisplay-picture control unit 22 changes the state into the INIT state(FIG. 12).

[0150] In the process of the MOVE state, in step S191, thedisplay-picture control unit 22 executes calculation of the motion basedon the following expression:

Motion=Motion Vector/Frame Rate

[0151] Next, proceeding to step S192, the display-picture control unit22 separates a picture corresponding to the viewpoint information fromthe viewpoint-information input unit 21, and outputs and displays thepicture on the picture output unit 25. In step S193, the display-picturecontrol unit 22 determines whether the viewpoint has been moved. If ithas been moved, the display-picture control unit 22 returns to step S191and continuously executes the subsequent steps (FIG. 12). In step S193,if it is determined that the viewpoint has not been move, thedisplay-picture control unit 22 proceeds to step S194 and changes thestate into the READY state.

[0152] An idea that only a portion being actually viewed is transmittedcan be applied also to the case of constituting a car navigation systemusing the Internet. In this case, by storing map information of theentirety in the form of pictures of JPEG 2000, and performing divisioninto tiles in units of, for example, degrees of latitude and longitude,a control method for the system can be handled similarly to distributionof partial omnidirectional pictures.

[0153] As described above, the picture receiving apparatus 2 calculatesthe margin of a portion which is actually output (displayed) by thepicture receiving apparatus 2 and which corresponds to a portion of apicture transmitted from the picture transmitting apparatus 1 to thepicture receiving apparatus 2. However, the calculation may be executedby the picture transmitting apparatus 1. In this case, based on theviewpoint information from the picture receiving apparatus 2, and thecalculated margin, the picture transmitting apparatus 1 determines andextracts from the omnidirectional picture a portion to be transmitted tothe picture receiving apparatus 2, and transmits the portion to thepicture receiving apparatus 2.

[0154] At this time, the picture transmitting apparatus 1 can determinethe size of the picture for transmission by measuring a response timewith RTCP. Also, the picture receiving apparatus 2 can determine andnotify the picture transmitting apparatus 1 of the size of the picture.

[0155] In addition, at this time, before initiating a session, in aset-up mode, or during a session, for example, by using RTSP, both thepicture transmitting apparatus 1 and the picture receiving apparatus 2must mutually obtain agreement about the moving speed of the viewpoint.

[0156] The above consecutive processing can be executed by hardware, butmay be executed by software. When software is used to execute theconsecutive processing, programs constituting the software are installedfrom a network or a recording medium into a computer built intodedicated hardware or into, for example, a multi-purpose personalcomputer in which by installing various programs, various functions canbe executed.

[0157] The types of the recording medium, not only includes magneticdisks (including floppy disks), optical disks (including CD-ROMs andDVDs), magneto-optical disks (MiniDiscs), and package media including asemiconductor memory, but also include ROMs and hard disks containingprograms.

[0158] In this Specification, steps constituting a program recorded onthe recording medium include, not only processes which aretime-sequentially performed in accordance with described order, but alsoprocesses which are executed in parallel or separately if they are notalways performed time-sequential order. In this Specification the systemrepresents the entirety of an apparatus including a plurality of units.

[0159] As described above, according to the present invention, a picturecan be distributed from a transmitting apparatus to a receivingapparatus. Also, even if the viewpoint is moved, the receiving apparatuscan output and display, in real time, a picture corresponding to the newviewpoint.

[0160] It should be understood that various changes and modifications tothe presently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present invention andwithout diminishing its intended advantages. It is therefore intendedthat such changes and modifications be covered by the appended claims.

1. A picture distribution system comprising: a receiving apparatus; anda transmitting apparatus in communication with said receiving apparatusfor distributing omnidirectional pictures to said receiving apparatusthrough a network; said transmitting apparatus including, firstreceiving means for receiving viewpoint information from said receivingapparatus; extracting means for, based on the received viewpointinformation, extracting a predetermined area as a first picture from anomnidirectional picture to be sent to said receiving apparatus; andfirst transmitting means for transmitting, to said receiving apparatus,unit regions obtained by dividing the first picture; and said receivingapparatus including: acquiring means for acquiring the viewpointinformation; second transmitting means for transmitting the acquiredviewpoint information to said transmitting apparatus; second receivingmeans for receiving the first picture transmitted from said secondtransmitting apparatus; storage means for storing the first picturereceived by said second receiving means; and output means for outputtinga second picture as a, predetermined area extracted from the storedfirst picture, the second picture corresponding to the acquiredviewpoint information.
 2. A picture distribution method for distributingan omnidirectional picture from a transmitting apparatus to a receivingapparatus by a network, comprising the steps of: a first receiving stepfor receiving viewpoint information from said receiving apparatus; anextracting step for, based on the received viewpoint information,extracting a predetermined area as a first picture from anomnidirectional picture to be distributed to said receiving apparatus; afirst transmitting step for transmitting, to said receiving apparatus,unit regions obtained by dividing the first picture; an acquiring stepfor acquiring the viewpoint information transmitted to said transmittingapparatus; a second transmitting step for transmitting the acquiredviewpoint information to said transmitting apparatus; a second receivingstep for receiving the first picture transmitted from said transmittingapparatus; a storage step for storing the first picture received in saidsecond receiving step; and an output step for outputting a secondpicture as a predetermined area extracted from the stored first picture,the second picture corresponding to the acquired viewpoint information.3. A picture distribution apparatus for distributing omnidirectionalpictures to a receiving apparatus through a network, comprising:receiving means for receiving viewpoint information from said receivingapparatus; extracting means for, based on the received viewpointinformation, extracting a predetermined area as a picture from anomnidirectional picture to be distributed to the receiving apparatus;and transmitting means for transmitting, to said receiving apparatus,predetermined unit regions obtained by dividing the picture.
 4. Apicture distribution apparatus according to claim 3, wherein, based on amotion vector included in the received viewpoint and a transmissiondelay of said network, said extracting means determines the nextpredetermined area to be extracted from the omnidirectional picture. 5.A picture distribution apparatus according to claim 3, furthercomprising calculating means for calculating the margin of apredetermined area which is output by said receiving apparatus inresponse to the viewpoint information, and which corresponds to thepredetermined area transmitted to said receiving apparatus by saidtransmitting means, wherein, based on the received viewpoint informationand the calculated margin, said extracting means extracts thepredetermined area as the picture from the omnidirectional picture.
 6. Apicture distribution method for a picture distribution apparatus fordistributing an omnidirectional picture to a receiving apparatus by anetwork, comprising the steps of: receiving viewpoint information fromsaid receiving apparatus; extracting, based on the received viewpointinformation, a predetermined area as a picture from the omnidirectionalpicture to be distributed to the receiving apparatus; and transmitting,to said receiving apparatus, predetermined unit regions obtained bydividing the picture.
 7. A recording medium containing acomputer-readable program for a picture distribution apparatus fordistributing an omnidirectional picture to a receiving apparatus, theprogram comprising the steps of: receiving viewpoint information fromsaid receiving apparatus; extracting, based on the received viewpointinformation, a predetermined area as a picture from an omnidirectionalpicture to be distributed to the receiving apparatus; and transmitting,to said receiving apparatus, predetermined unit regions obtained bydividing the picture.
 8. A program for a computer controlling a picturedistribution apparatus for distributing an omnidirectional picture by anetwork, said program causing the computer to execute the steps of:receiving viewpoint information from said receiving apparatus;extracting, based on the received viewpoint information, a predeterminedarea as a picture from the omnidirectional picture to be distributed;and transmitting, to said receiving apparatus, predetermined unitregions obtained by dividing the picture.
 9. A picture receivingapparatus for receiving an omnidirectional picture distributed from atransmitting apparatus through a network, said picture receivingapparatus comprising: acquiring means for acquiring viewpointinformation; transmitting means for transmitting the acquired viewpointinformation to said transmitting apparatus; receiving means forreceiving a first picture as a predetermined area extracted from theomnidirectional picture, the first picture being transmitted from saidtransmitting apparatus; storage means for storing the received firstpicture; and output means for outputting a second picture as apredetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information.
 10. Apicture receiving apparatus according to claim 9, further comprisingcalculating means for calculating the margin of the second picture whichcorresponds to the first picture, wherein, based on the calculatedmargin, said transmitting means transmits the viewpoint information tosaid transmitting apparatus.
 11. A picture receiving method for apicture receiving apparatus receiving an omnidirectional picturedistributed from a transmitting apparatus through a network, saidpicture receiving method comprising the steps of: acquiring viewpointinformation; transmitting the acquired viewpoint information to saidtransmitting apparatus; receiving a first picture as a predeterminedarea extracted from the omnidirectional picture, the first picture beingtransmitted from said transmitting apparatus; storing the received firstpicture; and outputting a second picture as a predetermined areaextracted from the stored first picture, the second picturecorresponding to the acquired viewpoint information.
 12. A recordingmedium containing a computer-readable program for a picture receivingapparatus for receiving an omnidirectional picture distributed from atransmitting apparatus by a network, the program comprising the stepsof: acquiring viewpoint information; transmitting the acquired viewpointinformation to said transmitting apparatus; receiving a first picture asa predetermined area extracted from the omnidirectional picture, thefirst picture being transmitted from said transmitting apparatus;storing the received first picture; and outputting a second picture as apredetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information.
 13. Aprogram for a computer controlling a picture receiving apparatus forreceiving an omnidirectional picture distributed from a transmittingapparatus by a network, said program causing the computer to execute thesteps of: acquiring viewpoint information; transmitting the acquiredviewpoint information to said transmitting apparatus; receiving a firstpicture as a predetermined area extracted from the omnidirectionalpicture, the first picture being transmitted from said transmittingapparatus; storing the received first picture; and outputting a secondpicture as a predetermined area extracted from the stored first picture,the second picture corresponding to the acquired viewpoint information.